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There can be no doubt that digital technology has found its way into the mainstream of the education 
field. However, educators today still struggle with how to best incorporate technology into their instruc- 
tion to promote student learning. One promising approach, blended instruction, combines online 
instruction with in-person learning activities. Such blended strategies have become increasingly popular, 
partly due to their success in combatting student disengagement in many school settings. We can 
expect continued growth in the adoption of blended and other technology-enhanced instructional ap- 
proaches in the years ahead. 


Research suggests that blended instructional formats are generally as effective, or more effective, than 
traditional instructional formats at improving student achievement (Shen, Wang, & Pan, 2008; Kliger 

& Pfeiffer, 2011; Moore & Gilmartin, 2010; Means, Toyama, Murphy, Bakia, & Jones, 2009; Heterick & 
Twigg, 2003). However, much of this literature focuses on higher education; there is limited research on 
blended instruction’s impact on achievement in K-12 contexts (Means, et al., 2009). While one study 
explores how the integration of technology in secondary classrooms positively affects student engage- 
ment (Pierce, Stacey, & Barkatsas, 2007), research on blended learning’s effect on student motivation 
has also been largely limited to undergraduate and adult populations. 


To address this gap in research, we conducted a mixed method, quasi-experimental study with a diverse 
set of high school students that examines the impact of a specific model of blended instruction on stu- 
dent engagement, skills acquisition, and achievement in science. 
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BLENDED INSTRUCTION 2 


Scope 


Our study was completed within the context of a larger, 
longitudinal research study in 12 urban high schools in 
Connecticut and Massachusetts. As such, our examination 
is multifaceted and sits within a larger body of research 
that examines the affective and cognitive impact of blended 
instruction on students in grades 9-12. 


We use a mixed methods approach to examine the impact 
of the STEM21 Academy’s blended instructional approach 
on 9th-grade students’ 1) engagement in science and math- 
ematics learning, 2) acquisition of 21st century and inquiry 
skills, and 3) academic achievement in science. Addition- 
ally, the study examines the impact that blended instruction 
practices had on underserved students in particular. 


We began with three quantitative questions, comparing 
9th-grade students who had participated in the STEM21 
Academy for one year with a control group that had not 
participated in the program. We asked: 


e Atthe end of 9th-grade, do STEM21 students dem- 
onstrate higher levels of engagement in science and 
mathematics than their peers? 


¢ Do they demonstrate higher levels of 21st century and 
inquiry skills acquisition? 

¢ Do they demonstrate greater achievement in science 
on the TerraNova science assessment? 


The qualitative portion of the study further elucidates these 
quantitative findings and consists of two components. In the 
first, we look at impact, asking: 


e What specific components of this blended instructional 
approach resulted in increased student engagement, 
21st century skills acquisition, and science achieve- 
ment? 


¢ What other contextual factors influenced the degree of 
impact of student learning? 


e Was this blended instruction model more successful 
for some students than others? What was its impact on 
underserved students in particular? 


The second portion of our qualitative study examines fidelity 
of implementation, asking: 


¢ To what extent did the teachers and schools implement 
the program as originally intended by the developers? 


¢ How much variation in implementation fidelity was 
there across schools and classrooms? 
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Background 
THE IMPORTANCE OF PEDAGOGY 


Advances in technology in the past several decades have 
created many new opportunities to integrate technology into 
classroom instruction (NCREL, 2005), but simply install- 

ing computers in schools does little to improve learning 
outcomes (Dror, 2008). The manner in which a teacher 
incorporates technology into instruction often reflects his 

or her pedagogical perspective (Henessy, Deaney, Ruthven, 
& Winterbottom, 2007; Niederhauser & Stoddart, 2001; 
Reeves, 1997), and that perspective greatly influences how 
successful students will be in technology-enhanced learning 
environments (Hartley & Collins-Brown, 1999; Purvis, et al., 
2011; Williams, 2002). 


National research consistently demonstrates that student- 
centered (or constructivist) instructional approaches more 
effectively develop both traditional and higher-order thinking 
skills than more didactic (or transmissive) forms of instruc- 
tion (Resnick & Zurawsky, 2007; Pink, 2009; Darling- 
Hammond, 2010). Student-centered approaches have been 
associated with enhanced student motivation and engage- 
ment and with more effective decision-making by teachers 
as they select appropriate technologies to enhance learning 
(Krueger, Boboc, Smaldino, Cornish, & Callahan, 2004; 
Venezky, 2004). 


Student-centered pedagogy emphasizes active learning 
through discovery and collaboration. In student-centered 
classrooms, students solve real-life problems, participate 

in group projects, investigate solutions to research ques- 
tions, and reflect on their thoughts and actions (Jonassen & 
Land, 2012). Student-centered, or constructivist, strategies 
support the transfer and retention of knowledge by offering 
students opportunities to study examples in context, explore 
underlying principles, make choices about their learning, 
receive timely feedback, and reflect in writing. Educational 
technology that is influenced by constructivist theories 

often emphasizes the discovery of connections between 
concepts, with open-ended applications that help students 
develop a deeper understanding of content (Niederhauser & 
Stoddart, 2001). 


In contrast, didactic learning models treat the teacher as 
the source of knowledge and students as passive recep- 
tacles (Jonassen & Land, 2012). In a typical example, the 
teacher might lecture at the front of the classroom, while 
students receive information espoused by the expert. There 
iS minimal interaction between the teacher and students 
or among students themselves. Transmissive educational 
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Digitally supported, student- 
centered pedagogy can make 
abstract concepts more tangible 
and help students appreciate 
practical future uses for those 
concepts in their lives. 


technologies are similarly didactic, emphasizing factual 
memorization and the practice of lower-order thinking skills 
(Niederhauser & Stoddart, 2001). Although such transmis- 
sive techniques may support foundational learning, they are 
generally not associated with the development of indepen- 
dent, self-directed learners. 


Student-centered learning approaches have psychologi- 

cal advantages, too; they have been shown to effectively 
combat student disengagement, a factor that contributes 
to the nation’s dropout problem. Student-centered environ- 
ments aim to meet each student at his or her current level 
of interest and ability. Generally speaking, classrooms that 
attend to each student’s engagement with learning seem 
to be effective at keeping students enrolled in school and 
increasing their academic achievement (Fredricks, Blumen- 
feld, Friedel & Paris, 2004; Azzam, 2007). Further, the type 
of deeper learning, critical thinking, and problem-solving 
skills developed through student-centered approaches may 
motivate students to take more ownership of their educa- 
tional pathways over the long term. 
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BLENDED INSTRUCTION AS A 
STUDENT-CENTERED APPROACH 


Blended instructional models can support a paradigm shift 
from teacher-directed to student-centered instruction by 
offering an online environment in which students can work 
through content at their own pace. Definitions of and ap- 
proaches to blended instruction vary widely, however. Singh 
(2003) describes blended instruction as a mix of various 
event-based activities, including face-to-face classrooms, 
live e-learning, and self-paced learning, while Zenger (2001) 
describes blended instruction as a completely integrated in- 
structional design, where the instructor-led and technology- 
based techniques fit logically together. 


For the purposes of this study, we will focus on blended 
instructional approaches that are also student-centered. In 
a student-centered, blended classroom, students use online 
technology to interact and collaborate with others, gather 
information, develop new knowledge, solve problems, and 
communicate ideas. Meanwhile, the teacher adopts the role 
of a knowledgeable “guide on the side,” supporting each 
student’s independent learning trajectory. Ideally, such 
blended learning environments encourage problem solv- 
ing, reflection, collaboration, and co-construction of new 
knowledge. 


Digitally supported, student-centered pedagogy can make 
abstract concepts more tangible and help students appreci- 
ate practical future uses for those concepts in their lives 
(Lee, 2010). Evidence suggests that the integration of pro- 
ficiency-based projects in a blended environment increases 
student engagement and achievement (DeGeorge-Walker, 
et al., 2010; Graham, 2006; Lou, et al., 2011; Wheeler, et 
al., 2010), as well as their ability to direct their own learning 
(Donnelly, 2010; Jeffries & Hyde, 2010). 


Stem21 Academy: 
PROGRAM OVERVIEW 


Science, Technology, Engineering and Mathematics 21 
(STEM21) Academy is a grades 9-12 program that uses a 
blended instructional approach to increase student engage- 
ment and achievement. The program’s primary mission 

is to integrate digital media skills into the context of a 
high-quality science and technology curriculum. STEM21’s 
school-within-a-school model creates cohorts of students 
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within a larger school who participate in a core set of 
courses together (Table 1). EDUCATION CONNECTION began 
implementing the STEM21 Academy model in Connecticut 
schools in 2000. 


In STEM21 courses, students solve real-world problems 
through self-directed and authentic learning tasks. 


Technology-mediated activities help students develop 
content knowledge and the 21st century creative-productive 
skills that support success in school, higher education, 

and work beyond school (see Figure 1). Technology is also 
leveraged to maximize student interactions and decrease 


Table 1: STEM21 Academy Scope and Sequence 


feelings of isolation. STEM21 faculty have opportunities to 
develop their capacity using blended instructional strategies 
through intensive summer learning conferences, ongoing 
in-class support and coaching, and face-to-face and virtual 
professional learning communities. 


STEM21 Academy’s blended instructional model includes 
four major components: technology-enhanced learning, 
experiential learning, digital portfolios, and proficiency as- 
sessments (see Figure 2). The following section describes 
each of four major components in detail. 


BO) ANIN 9TH-GRADE TOTH-GRADE 11TH GRADE 12TH GRADE 
Science Earth and Energy Biology21 Chemistry21 Physics21 
Essentials (E3) 
Technology Skills21 Digital Media & Movie Game Design & Capstone Experience 
Making Development 
or or 
Research, Design, & E-commerce 
Development Entrepreneurship 
Mathematics Algebra21 Geometry21 


Note: The numeral “21” indicates courses taught in a 21st century context (e.g., biology in the context of biotechnology, or chemistry 
in the context of material science). In addition to their STEM21 coursework, students take classes in other subject areas (e.g., English, 
social studies) as dictated by individual school’s course of study and graduation requirements. 


Figure 1: STEM21’s Priority Skills 
for 21st Century Success 
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Figure 2: Components of STEM21's 
Blended Instruction Model 
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PROGRAM COMPONENT 1: 
TECHNOLOGY-ENHANCED LEARNING 


STEM21’s curriculum combines face-to-face and computer- 
mediated activities with technology, providing options for 
customized learning and enhanced interpersonal interac- 
tions. A learning management system (or LMS) houses the 
online components of STEM21 courses; through this online 
platform, students access standards-aligned, scaffolded 
learning activities. Students use the LMS to post assign- 
ments, communicate with the teacher, participate in tutori- 
als and discussion forums, 

and complete assessments. 


The LMS supports differen- 
tiation by offering flexible, 
developmentally appropriate 
tasks. Students who move 

at a faster pace can access 
extension activities to expand 
and deepen their under- 
standing of new content. The 
teacher serves primarily as a facilitator of student learning, 
supporting students through messaging, online posts, and 
discussion forums, as well as one-on-one time in the class- 
room. Technology-integrated assessments allow teachers 

to monitor student progress and provide instant feedback 
and targeted support. Students can access resources at any 
time and tap into a support network as needed. 


The LMS provides several options for online student interac- 
tion, including asynchronous forums, blogs, and wikis, as 
well as synchronous chats, video conferencing, and virtual 
worlds. Each tool has unique benefits, and all are designed 
to promote deeper, more meaningful interactions among 
students in and out of the classroom. Within the virtual 
world, participants develop a customized, three-dimen- 
sional, self-concept (or avatar) that represents a multidi- 
mensional ‘‘malleable self’ (Jin, 2010). Avatars provide 
opportunities for real-time virtual collaboration. Such virtual 
experiences have been found to foster the types of social re- 
lationships that are critical to successful collaboration and 
learning (Ducheneaut & Moore, 2004; Seay, Jerome, Lee, & 
Kraut, 2004; LaBanca & Lorentson, 2013; Yee, 2006). 
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All of these experiences provide 
students with knowledge and 
expertise beyond what a single 

teacher could provide. 


PROGRAM COMPONENT 2: 
EXPERIENTIAL LEARNING 


Experiential learning activities engage students meaning- 
fully with the curriculum, often in settings beyond the class- 
room walls. In the experiential component of the STEM21 
curriculum, students visit authentic learning environments, 
including businesses and college campuses, where they 
have opportunities to interact with knowledgeable adult 
mentors. Through structured experiential meetings and 

an end-of-year Innovation Expo, students collaborate with 
professionals, present their work, 
and receive valuable feedback from 
mentors. Company tours provide 
opportunities to observe how 
technology and 21st century skills 
are used in practice. Because this 
project spanned multiple schools, 
experiential meetings had a value- 
add of being interdistrict whereby 
students collaborated, not only 
with industry professionals, but 
each other, creating interactions with considerable student 
diversity. 


Back in the classroom, students interact with mentors face- 
to-face and through online video chat services and forums. 
STEM21 also brings professionals into the classroom for 
interviews, guest lectures, and other interactive learning 
opportunities. All of these experiences provide students 
with knowledge and expertise beyond what a single teacher 
could provide. 


PROGRAM COMPONENT 3: 
DIGITAL PORTFOLIOS 


STEM21 Academy utilizes an open-source content manage- 
ment system for students to showcase their work digitally. 
A tool for reflection and sharing, the digital portfolio allows 
students to document and present their work to others. 
Each student has a personal web address, where they dis- 
play a portfolio of their own work. Portfolios enable students 
to demonstrate knowledge and skills, define their interests, 
focus on building particular expertise, communicate ideas 
and personal values, and celebrate growth. Students may 
share their site with peers, parents, prospective employers, 
and college admissions officers. 
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To guide students in selecting work for their portfolios, 
STEM21 Academy classes use a 21st century skills frame- 
work (see Figure 1 on page 5). For example, a Biology21 
student might select a lab report that demonstrates their 
understanding of diffusion and osmosis, classifying this 


assessment under the 21st century skill of problem-solving. 


The student would then provide a reflection to demonstrate 
how his or her problem-solving skills improved through the 
assignment. 


PROGRAM COMPONENT 4: 
PROFICIENCY ASSESSMENTS 


STEM21 Academy courses are made up of standards- 
aligned learning units that each explore a set of essential 
questions. In each unit, students perform a variety of learn- 
ing activities—e.g., face-to-face discussions and debates, 
readings, lab activities, and small projects—that build 
toward mastery of the course objectives and support them 
to become more independent, self-directed, and collabora- 
tive learners. 
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Each unit concludes with a capstone Unit Performance 
Assessment (UPA). These authentic, project-based assess- 
ments provide diverse opportunities for students to demon- 
strate mastery of course objectives according to a four-point 
rubric. UPAs are intentionally open ended and loosely de- 
fined (Jonassen, 1997), allowing for student choice and self- 
direction, while measuring essential learning outcomes. The 
products can easily be embedded into a digital portfolio. 


Every course culminates in a Challenge Project, an extended 
project that derives from an essential question or challenge 
and serves as a final demonstration of proficiency. For each 
project, students engage in exploratory activities, gather 
resources, determine a solution, implement that solution, 
reflect, assess their Success, and share the results. These 
extended, team-based projects require students to integrate 
course content with relevant skills through technical writing, 
research, computer-assisted design, team collaboration, 
field experiments, storytelling, and/or web-based presenta- 
tion. 
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Figure 3: Sample Unit Performance Assessment 


COURSE: EARTH AND ENERGY ESSENTIALS 


Grade level: 9 
Unit: Natural Resource Use and Environmental Impact 


Assignment: 


Your environmental consulting firm has been hired by the state’s Department of Environmental 
Protection (DEP) and has been tasked with researching one of four important topics related to the 
environmental effects of natural resource use. Each team in the firm must conduct research and 
create an educational piece designed to inform the general public about: 


¢ municipal solids waste (garbage) and its disposal (e.g., landfills, incineration) 
¢ pollutants generated by industry and agriculture (e.g., mercury, phosphates, nitrates.) 


* pollutants generated by energy production in power plants and automobiles (e.g., SO,,, 
Cco,, NO.) 
2 x 


e land use (e.g., housing development, transportation, mining) 


Research: 
Research and identify a list of natural resources and effects that are connected to the topic. 


Research and identify a variety of options for minimizing the environmental effects of a natural 
resource of your selection. Include some practical suggestions for how we can balance our need for 
natural resources and problems caused by their use. 


Product: 


Choose how you would like to present your findings to the public. You may: 
¢ create an educational pamphlet that could be distributed in the local community. 


¢ create an informational video, podcast, or public service announcement that could be 
placed on the DEP website or aired on local TV stations. 


¢ develop presentation materials (i.e., PowerPoint, Prezi) that can be used by the DEP at 
community meetings. 


¢ design another method of presentation. 


Whichever format you choose, your group should include specific statistics, facts and figures when- 
ever possible (include citations). Your project will be evaluated using a rubric to indicate the quality 
of (a) research, (b) information, (c) presentation, and (d) teamwork. 


A model depiction of an instructional year is provided in Figure 4 on next page. 
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Figure 4: Stem21 Academy Model Program Implementation 


Fall Semester: Individual Spring Semester: Team 
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Content knowledge in core academic disciplines: e.g., Science: Earth and Energy Essentials 


Curriculum units in Learning Management 
System support completion of 3 Unit Per- 
formance Assessments (UPA) to proficiency. 
UPAs are designed to allow students to dem- 


Pre-post 
formative 
assess- 


Team show- 
case of Chal- 
lenge Projects 


Team-based curriculum: units differentiated by area 
of responsibility, expertise, and department. Students 
collaborate to create a product, service, or technology 


Pre-post 
formative 
assessment, 


Computer-mediated / online learning 


Face-face classroom work 


Experiential learning 


ment, 
surveys, 
writing 


Orientation 
to Digital 
Portfolio 


Orientation 
to program 
including 
challenge 
project, 
blended 
learning, 
LMS, digital 
portfolio, 
experiential 
learning, 
career 
pathways, 
student 
success 
plans 


Authentic 
assessment 
of student 
work by a 
panel of 
business, 
industry, and 
academic 
profession- 
als 


onstrate proficiency to standards and unit’s 
essential questions. Periodic standardized 
assessments. Documentation and reflection 
of learning in digital portfolio. 


UPA 1 UPA 2 UPA 3 


that solves an authentic problem. 


surveys, 
writing; 
reflection of 
learning 


Asynchronous and synchronous tools for collaboration: forums, wikis, blogs, discussion boards, 3D virtual worlds, Internet video calls, 


digital media products 


Student selects work for submission to portfolio, reflects on learning, and receives feedback from teacher, peers, and adults 


(parents/mentors). 


suet Students demonstrate profi- 


ciency to standards for UPAs 
and development of 21st 
century skills: 


Student 2 


information literacy 
collaboration 
communication 
innovation/creativity 
problem solving 
responsible citizenship 


Student 3 


Student 4 


Student 5 


Student 6 


Workplace tours, meetings, guest 
speakers, classroom visits 


Onsite 
Meeting 1 


Onsite 
Meeting 2 


Virtual com- 
munication: 
internet video 
calls (e.g., 
Skype); 3D 
virtual worlds 
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Students 1, 2, 3, who have demonstrated profi- 


ciency on UPAs form the class team and begin to 


develop solutions to the program challenge 


Student 4 joins team 
upon demonstration 
of proficiency 
demonstration of proficiency aS 


Student 5 joins team upon 


Student 6 continues to work on individual pro- 
ficiencies and participates in team activities in 
consultation with teacher 


Online and onsite mentoring by business, industry, 
and academic professionals. Biweekly progress 
reports and responses. 


Onsite 
Meeting 3 


Virtual com- 
munication: 
Internet 
video calls; 
3D virtual 
worlds 


Authentic 
assessment 
of student 
work by a 
panel of 
business, 
industry, and 
academic 
profession- 
als using 
standardized 
guidelines 
and rubrics 


Showcase 
and public 
display of 
student work 
evaluated by 
business and 
academic 
professionals 


Students 
debrief and 
reflect on 
learning 
leveraging 
technology 
and 

21st century 
skills 


Further 
mentoring 
and intern- 
ship op- 
portunities 
explored 
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STEM21 in Practice 
A CASE STUDY FROM THE FIELD 


At the start of 9th-grade, Miguel was a low-performing Eng- 
lish language learner attending a large urban high school. 
All three of his STEM21 Academy teachers noticed that 
Miguel consistently chose to do the least work possible and 
barely passed his courses as a result. He lacked motivation, 
had limited subject-area knowledge, and struggled signifi- 
cantly with teacher-directed instruction. When his teachers 
introduced the annual Challenge Project, Miguel began to 
change his behavior. He participated more in class, com- 
pleted assignments at home more regularly, and began to 
ask more questions. 


The project asked students to “create and implement a new 
product or service that addresses the theme of Responsible 
Design.” Miguel had to work with his team members to: 


¢ identify a specific issue or topic. 


e investigate the topic through research, experimenta- 
tion, and data analysis. 


e develop an innovative solution (product or service) that 
addresses the issue. 


¢ demonstrate application of the product or 
service. 


¢ document and communicate the project using digital 
media. 


Additionally, for their Earth and Energy Essentials class, 
students were also asked to integrate the conservation or 
responsible use of one or more natural resources into the 
project. 


Miguel and his team felt it was important to choose a 
problem relevant to their community, and they settled on 
mosquitos. They decided to create the "Water Hornet," 
named after their school mascot - a solar-powered, battery- 
operated device capable of creating motion in still water to 
prevent mosquitoes from laying eggs. 


With a project of this scale, the team needed to departmen- 
talize. Daniel, their math teacher, was surprised to see that 
Miguel showed great interest in creating a sophisticated 
spreadsheet that could be used to input data and connect 
to state data on mosquito distribution to generate predic- 
tions. The formulae required a keen understanding of the 
point-slope form of a linear equation, a concept Miguel 
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had previously struggled to master. Now, he rose to the 
challenge, becoming proficient with point-slope form and 
developing a deep understanding of concepts that his team 
applied to their project. 


His science and technology teachers noticed that Miguel 
began to take on a leadership role when he was in their 
classes too. With adept tactile skills, he helped to build a 
prototype device and made significant contributions to the 
project’s website. All three teachers agreed that Miguel was 
consistently demonstrating high-quality work and leadership 
among his peers. 


This project represented a major shift for Miguel. With 
an assignment that felt relevant and a new team-based 
context, he finally felt motivated to take ownership of his 
learning. 


Study Design 
QUANTITATIVE IMPACT STUDY DESIGN 


Participants: For our intervention group, we recruited 9th- 
grade students who had registered for STEM21 Academy 
programs in 11 schools (10 in Connecticut, one in Massa- 
chusetts). We recruited a comparison group from students 
enrolled in comparable 9th-grade science courses in the 
same schools who had similar demographic backgrounds 
to students in the intervention group. All together, our initial 
sample included 500 9th-grade students (233 in the inter- 
vention group and 277 in the comparison group). 


Our baseline analyses demonstrated that STEM21 Academy 
and comparison groups were not equivalent at the outset in 
terms of their 21st century and inquiry skills and TerraNova 
science scores. (They were, however, equivalent in terms of 
engagement.) Therefore, we created matched subsamples 
from our initial participant set to measure the impact of the 
STEM21 program on science achievement and skills growth. 
Subsample 1 included 216 students from two cohorts 
(2011-2012 and 2012-2013 school years) with equivalent 
baseline scores on the TerraNova science achievement 
assessment. Subsample 2 included 128 students from the 
2012-2013 cohort with similar baseline scores on our skills 
and engagement assessments. (Subsample 1 could not 

be used to assess 21st century skills and engagement, as 
those data were not collected in the 2011 to 2012 school 
year.) More details about the creation of the matched 
subsamples are described in the analytic strategies section 
below. 
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Data collection process: We collected data twice: once 


at the beginning of 9th-grade and again at the end of the 


9th-grade. Each time, participating students took a 40-min- 


ute online survey that included items from three different 
survey instruments. Then, within a week or so, students 


took a 40-minute TerraNova Science assessment in paper- 


pencil format. We received parental consent and student 
assent prior to the survey and TerraNova administration. 


Instruments and measures: To measure program 
impact, we used three instruments: 


1. Academic engagement was measured using an 
adapted version of the Academic Engagement instru- 
ment (Weinstein, Schulte, & Cascallar, 1983), com- 
bined with seven items from the motivation scale of 


21st century skills were measured using a survey 
instrument incorporating six 21st century skills and 14 
inquiry skills derived from NCREL’s enGauge standards, 
the ISTE NET-S standards, and the Partnership for 21st 
Century Skills framework. The researchers developed 

a self-report scale to examine students’ strengths in 
information literacy, collaboration, communication, 
innovation and creativity, problem solving, and respon- 
sible citizenship. Students were asked to rate their 
level of perceived skill on 12 tasks and behaviors (e.g., 
“Evaluating the validity of data or evidence collected 
from a STEM product,” or “Determining an innovative 
solution to a STEM challenge”), using a seven-point 
Likert scale (1=“extremely poor;” 7=“excellent”). The 
reliability of this scale was .92 (Cronbach's alpha) for 
pre and post data. 


the Learning and Study Strategies Inventory. Students 
reported on the degree to which they stayed on top 

of class assignments, came to class prepared, set 
high expectations for themselves, and persisted in the 
completion tasks and achievement of good grades. 
We modified the original items to specify the examined 
subjects. For instance, the original item “Even if lam 
having difficulty in my course, | can motivate myself to 
complete the work,” was adapted to “Even if | am hav- 
ing difficulty in a science course, | can motivate myself 
to complete the work.” The seven items were tested in 
both science and mathematics domains, for a total of 
14 items. The reliability of the survey was .86 for both 
pre and post data. (A scale is considered reliable when 
Cronbach’s alpha is higher than .7). 


3. Science achievement was measured using the 9th- 
grade TerraNova assessment, a standardized, norm- 
referenced test that assesses student understanding 
of science theory and application with emphasis on 
understanding core concepts and applying scientific 
inquiry skills, including the scientific method. The 
TerraNova subtests are considered reliable instru- 
ments; the reliability alpha for the grade 9 test is .82, 
as reported in the Technical Bulletin compiled by CTB/ 
McGraw-Hill. 


We also collected student demographic data, including par- 
ticipants’ gender, race/ethnicity, free/reduced lunch status, 
special education status, and English language learning 
status. We then created a combined variable represent- 


Table 2. Description of Matched Samples 


DEMOGRAPHIC SUBSAMPLE 1 (%) SUBSAMPLE 2 (%) 


CHARACTERISTIC 


Ethnicity African-American 14 24 


Asian-American 3 3 

Latino 16 23 

Pacific Islander ‘il 0 

White 64 48 

Other 8 3 

ee ee a a cn se! 

Other characteristics Special education tf 3 

English language learner all 
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ing a student’s “underserved” status—with racial/ethnic 
minorities, females, and low-income students defined as 
underserved. We also noted each school’s SES composition 
(less than or more than 50 percent receiving free/reduced 
lunch). The students’ underserved status, special needs sta- 
tus, and their cohort and schools’ SES composition served 
as covariates in the analytic model. 


Analytic strategies: We used a hierarchical linear model- 
ing (HLM) to estimate the impact of the STEM21 Academy 
on student engagement, 21st century skills, and science 
achievement after one year, controlling for baselines tests, 
cohort, special education status, students’ underserved 
status, and school SES composition. Unlike other regression 
techniques, HLM simultaneously accounts for variations 
between schools and individuals, even over time. 


QUALITATIVE STUDY DESIGN 
PART 1: TEACHING AND LEARNING 


We conducted semi-structured interviews with 9th-grade 
students from each of the participating schools to better 
understand the impact of blended instructional strategies on 
their experience as learners. We interviewed an additional 
three 10th-grade students from each participating school 
and held focus groups with STEM21 teachers to confirm and 
further elucidate interview results. 


Interview questions focused on students’ and teachers’ 
perceived impact of the four core components of STEM21 
Academy’s blended instructional approach: technology-en- 
hanced learning, experiential learning, digital portfolios, and 
proficiency assessments. Participants were first asked if any 
portion of the course implementation had an especially sig- 
nificant impact on their learning. The next set of questions 
addressed each of the four components directly. Follow-up 
questions were used to clarify or further develop responses. 


We developed our interview protocol using a taxonomy 

of ethnographic questions and vetted questions through 

a peer auditing process, exploratory focus groups, and a 
series of 15-minute pilot interviews with students in spring 
2012. Results of these tests were used to revise and im- 
prove the final interview protocol. 


During spring 2013, we conducted 15-minute interviews 
with three 9th-grade students from each of the 12 partici- 
pating schools (a total of 36 interviews). All students were 
new to the STEM21 Academy program that year. To sup- 
port and triangulate our data, we conducted 12 additional 
15-minute interviews with 1Oth-grade students; these 
students had been in the program for two years and could 
therefore provide a longer-term perspective on the program. 
In addition, we held six focus groups with STEM21 Academy 
teachers in 2012 and 2013. 


All interviews were recorded and transcribed verbatim, and 
interviewees had an opportunity to correct transcripts for 
accuracy. Field notes were used to maintain an audit trail. 
Interviews were conducted until we reached data satura- 
tion. 


We conducted a content analysis of interview data, using 
NVivo software to search for word and phrase patterns and 
categories. Recurring regularities in the data were organized 
into the construct categories, and that data was triangulat- 
ed by data source (teachers and students) and data method 
(interviews and observations). 


We conducted semi-structured interviews with 9th-grade students 
from each of the participating schools to better understand the impact of 
blended instructional strategies on their experience as learners. 
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Table 3: STEM21 Academy Implementation Fidelity Indicators 


TEACHER PROFESSIONAL 


DEVELOPMENT INDICATORS 


GRADE-LEVEL INDICATORS 


PROGRAM-LEVEL INDICATORS 


¢ Participation in Summer Academy 
(Year 1: all teachers; Years 2-4: first- 
year teachers only) 


e Participation in two coaching 
sessions annually (Years 2-4) 


¢ Participation in one end-of-year 
reflection session (Years 2-4) 


Percent of students completing the 
digital portfolio 


Percentage of curriculum units 
completed by teacher 


Percentage of students attending 
three experiential meetings and the 
Innovation Expo 


Participation of students completing 
five forums in one technology class 


Summer Institute: 28 hours of 
quality professional development 
including 21st century skills 


-Or- 


Make-up session: 6 hour session 
for individuals unable to attend the 
Summer Institute 


Participation of each teacher in two 
coaching sessions (Years 2-4) 


and one science class 


¢ Participation in one end-of-year 
reflection session (Years 2-4) 


e Atleast 51% of students in study 
from underrepresented groups as 
defined by gender, minority status, 
and SES 


¢ Expected number of students 
participating in high school program 
(Calculated at 20 students/grade/ 
school/year) 


e Eight courses in 12 schools 
implemented as planned (Year 4) 


¢ Six courses articulated with College 
of Technology (Year 4) 


Note: Changes in specific development indicators over time are identified in parenthesis. 


PART 2: FIDELITY OF IMPLEMENTATION 


We assessed implementation fidelity in the 12 participating 
schools through the ongoing collection of: (a) professional 
development data records; (b) attendance records, portfolio 
grades, Unit Performance Assessments, and records of on- 
line forum use; and (c) professional development agendas 
and observations. We developed a metric, ranging from “low 
fidelity” to “high fidelity,” using these data sources to assess 
the degree to which the blended instructional program was 
implemented as intended at the individual teacher level, 
within the grade level, and across the program. 


We developed categorical indices (e.g., teacher attendance 
in summer professional development, students completing 
the digital portfolio) to establish implementation bench- 
marks for each treatment classroom, grade-level, and 
school. We assigned weights to each indicator appropriate 
to the relevance and importance of that factor in program 
implementation, and used the combined indicator scores to 
generate a construct-level score. Each construct score was 
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categorized into levels of implementation fidelity (low, mod- 
erate, high), with our threshold for “adequate implementa- 
tion” of most categories set at completion of 65% or more 

of relevant activities. 


Our research team also conducted observations of eight 
9th-grade classrooms, using the Electronic Quality of Inquiry 
Protocol (EQUIP) and the Classroom Practices Record (CPR) 
as tools for organizing observations. The EQUIP is a four- 
point observational rubric designed to measure the number 
of instances and quality of inquiry-style instruction. The 
CPR measures higher-order thinking and questioning in the 
classroom. Each 9th-grade classroom was observed two 
times, with preliminary data obtained during spring 2012 
and comprehensive data collected during 2012-2013. 
These qualitative data were used to confirm our fidelity 
measures. 
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Findings 


QUANTITATIVE RESULTS: 


Impact Of STEM21 Practices On Student 
Engagement, 21st Century And Inquiry Skills, And 
Science Achievement 


At the end of 9th-grade, students who participated in the 
STEM21 Academy for one year demonstrated significantly 
higher scores on the TerraNova science assessment and 
marginally higher levels of 21st century skills than compari- 
son students. We found no statistically significant difference 
between student groups in terms of engagement in science 
and mathematics. Underserved students in the STEM21 
Academy performed better on the science assessment 

than underserved students in the comparison group. Small 
sample size prevented an analysis in engagement and 21st 
century skills for underserved students. Table 4 summarizes 
the quantitative results. 


STUDENT ENGAGEMENT 


At the end of 9th-grade, students in the STEM21 Academy 
group reported an average level of academic engagement 
of 5.03 (SD=1.06) on a 7-point scale (1=“not at all true of 
me”; 7=“very much true of me”). The average engagement 


level in the comparison group was 5.04 (SD=.90), indicating 
very similar degrees of engagement in both groups. Our sta- 
tistical analysis suggests that the STEM21 Academy did not 
engage students in science and math better than traditional 
classrooms. 


21ST CENTURY AND INQUIRY SKILLS 


At the end of 9th-grade, students in the STEM21 Academy 
sample and the comparison sample rated themselves an 
average of 5.37 and 5.07 respectively on our skills Survey, 
indicating that students from both groups believed they had 
acquired a mid-to-high level of 21st century skills. 


The score difference between the two groups was margin- 
ally significant (p =.06), taking into consideration other 
factors that might contribute to students’ 21st century skills 
acquisition (e.g., their baseline skills, student demograph- 
ics, and school SES composition), the STEM21 Academy 
group students scored .29 standard deviations higher on 
21st century skills than the comparison group. Accord- 


Table 4. Descriptive Statistics of Engagement, 21st Century and Inquiry Learning Skills, and 
TerraNova Science Measures 


95% 
CONFIDENCE 
INTERVAL 
Engagement Comparison 64 5.04 0.90 [4.82 5.27] 
Intervention 64 


21st Century skills | Comparison 


[4.77 5.29] 


[4.86 5.28] 


Intervention 


“Comparison | 116 


TerraNova Science 


: [5.16 5.59] 


| (674.75 692.59] 


Intervention 


100 


Underserved 90 


[687.54 704.06] 


1682 


| (67.54 693.48] 


TerraNova Science 47.59 

Comparison 

Underserved 75 693.75 44.21 [683.58 703.92] 
Intervention 
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Figure 4: Average self-reported student 
engagement and 21st century skills at the end 
of 9th-grade 
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ing to Lipsey, et al. (2012), the median effect size of high 
school intervention on student outcomes of specialized 
topic is .29 standard deviations, and the mean effect size 
is .34 standard deviations. In short, the STEM21 Academy 
was somewhat more successful in improving student 21st 
century skills than the comparison classrooms. (See Figure 
5 for details.) 


SCIENCE ACHIEVEMENT 


At the end of 9th-grade, students in the STEM21 Academy 
group scored higher on the TerraNova science test than 
students in the comparison group: a mean score of 695.80, 
compared with 683.67 (TerraNova scale scores range from 
O to 999). The result of statistical analysis confirmed that 
the difference between the two groups in science achieve- 
ment was Statistically significant (p <.05). Accounting for 
other factors, such as students’ demographic information, 
previous TerraNova scores, cohort and school SES com- 
position, the STEM21 Academy group students scored .25 
standard deviations higher on TerraNova science than the 
comparison group. The median effect size of a high school 
intervention on standardized test scores is .04 standard 
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Figure 5: Average TerraNova scale scores for 
students at the end of 9th-grade 
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deviations, and the mean effect size is .03 standard devia- 
tions (Lipsey, et al., 2012). Our results, therefore, suggest 
a meaningful effect of the STEM21 program on student 
achievement. 


UNDERSERVED STUDENTS 


Results for underserved students revealed similar patterns. 
That is, underserved students in the STEM21 Academy 
group scored higher on the TerraNova test than under- 
served students in the comparison group (a mean score of 
693.75, compared with a mean of 683.51). The difference 
between the two groups was Statistically significant (p <.05). 
Accounting for other factors, such as students’ previous 
TerraNova scores, student demographics, and cohort and 
school SES composition, we found a meaningful effect size 
of .23 (Cohen’s d). 


These results indicate that the STEM21 Academy program 
was better able to meet the learning needs of underserved 
students than traditional classrooms. (See Figure 5 for 
details.) 
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QUALITATIVE RESULTS: 


Perceived Impact and Fidelity of 
Implementation 


GENERAL PERCEPTIONS OF STEM21’S IMPACT 


When asked an open-ended question about the elements 
of the STEM21 Academy program that most influenced 
them as learners, the largest number of 9th-grade student 
responses (20%) emphasized how the program contributed 
to their success. Examples of the successful outcomes they 
discussed included increased engagement, a positive learn- 
ing experience, altered future intentions related to careers 
or academic interests, and increased achievement. By con- 
trast, very few (4.9%) student responses related to learning 
challenges, such as disengagement or a negative learning 
experience. These results indicate that STEM21 Academy 
students strongly believe the student-centered learning 
process supports their academic success. 


Interview data was coded and organized into the following 
thematic categories: overall learning success, improved 
engagement, and personal skill development. 


Overall learning success: Students said that their 
experience in STEM21 resulted in increased knowledge 
acquisition, enhanced skills in areas such as critical think- 
ing and communication, the development of independent 
learning habits, improvement in content retention, and an 
increase in confidence. These topics represent half (50%) 
of all responses in the overall perceptions category. Sample 
comments included: 


“The way that they teach you how to dig deeper 

and how you are the one that figures it out for 

yourself—you definitely remember—it becomes 
a lesson you learn forever.” 


“It really allows you to expand on your 
creativity. It gives you a much more, almost a 
more individualized learning.” 
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Improved engagement: The second most common cat- 
egory of response was related to different facets of engage- 
ment. Students mentioned overall enjoyment, fun, liking 

or loving the courses, increased interest in course content, 
creativity, appreciation of the up-to-date curriculum, com- 
mitment to further participation at the next grade level, and 
enthusiasm regarding the uniqueness of the non-traditional 
instructional approach. This category represents a third 
(32%) of all feedback. Sample comments included: 


“When you learn something new and it 
interests you so much, you want to learn 
more and more, and it’s a building block that 
compacts all your knowledge together and you 
can do things you never thought were possible.” 


“Really fun, hands-on stuff. It’s always 
changing. It’s a little bit of a challenge but if 
you are really into it, you'll get a lot out of it.” 


“In these courses, you go on and learn by 
yourself and you connect with other kids and I 
love the courses!” 
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Personal growth and skill development: The third 
most frequently occurring comments related to personal 

growth. Eighteen percent of responses related to this cat- 
egory. Comments addressed: 


e STEM21 Academy’s emphasis on team work and the 
use of interpersonal skills required for successful 
teamwork, 


¢ increased communication capabilities resulting from 
consistent personal and professional interactions, 

¢ enhanced friendships and a sense of “family” within 
the class environment, 

¢ development of a sense of interdependence with peers, 

¢ increased leadership ability, 


* appreciation for the presence of an open, non-judg- 
mental learning environment, and 


¢ increased student-teacher interaction. 
Sample comments included: 


“My confidence has built up tremendously over 
the course of this year and it has been amazing.” 


“It’s really hands-on and teaches you to be more 
independent in your learning. It is very different 
from what I've had before.” 


“You got to be really close with a group of 
students so you get a nice bond. Yet you can still 
be independent and grow at your own level.” 


Additional impressions: |n addition to the three primary 
categories, students cited a variety of other successes that 
they attributed to STEM21 Academy’s student-centered 
approach. We found a consistent pattern of overall student 
enthusiasm toward the blended instructional approach, 
with the majority of students affirming their commitment 

to continued participation in the program. We heard such 
comments as: 


“I would describe these courses as more modern 
learning...It’s more on technology and it’s not 
really old school learning.” 


“You'll learn to use teamwork and 
communication—it is a great course if you want 
to work for a big company in the future.” 
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“I think the whole process of learning how to fit 
in and work with what you have was a good 
experience for us because we are going to be 

thrown into the new world this same way.” 


A number of respondents indicated that they considered the 
blended curriculum to be a highly relevant learning method 
for the 21st century, arguing that individuals in their age 
group would be heavily dependent on technology skills for 
future college and job opportunities. Additionally, some 
students stated that participation in these courses would 
give them a competitive advantage in the college or career 
market over students participating only in more traditional 
courses. 


Perceived challenges: Although the breadth of re- 
sponses strongly suggested that the majority of 9th-grade 
STEM21 Academy students planned to remain enrolled in 
10th-grade, a small number of participants (approximately 
10%) were less certain about the success of their learning 
experience in the first year of the STEM21 Academy pro- 
gram. Individuals who indicated a lack of interest in project 
participation generally attributed this lack of interest to the 
use of technology. Additionally, a small number of students 
expressed frustration with the blended curriculum, stating a 
preference instead for more traditional learning that places 
less emphasis on technology-driven assignments and out- 
comes. 


Students also described various implementation or learn- 
ing challenges, including challenges using the learning 
management system (LMS) to complete the digital portfolio, 
participate in forums, and complete other online assign- 
ments. Students discussed inconsistent performance of the 
LMS, sometimes attributed to system incompatibility or Wi- 
Fi access issues in their school. Other students perceived 
the LMS as too difficult to understand. Students with these 
concerns expressed a desire for more teacher-directed 
instruction and felt a lack of written guidelines exacerbated 
their technological challenges. These described a need for 
more technology support. 


General perceptions of teachers and 10th-grade 
students: The overall perspectives of 9th-grade students 
were widely substantiated by teachers and their 10th-grade 
peers. Teacher focus groups emphasized how blended 
instruction resulted in more student freedom and indepen- 
dence and reported an increase in students’ learning skills. 
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Comments from 10th-grade students suggested that they 
generally engaged in the program with a higher level of 
confidence and understanding than their 9th-grade counter- 
parts. A number of 10th-grade students reported that their 
knowledge and confidence had increased since 9th-grade. 
These students attributed this increase to the development 
of a better working knowledge of curriculum expectations 

in their second year. Tenth-grade students were also more 
likely to describe potential long-term benefits of the program 
than were their 9th-grade peers. When questioned about 
learning outcomes, the large majority (88%) of 1Oth-grade 
respondents had a positive perception of the impact of 
blended instruction on both their personal growth and suc- 
cess with learning. 


Tenth-grade students expressed fewer program challenges, 
saying many challenges had been resolved during their first 
year. Among the 22% of 10th-graders who identified chal- 
lenges, only a few focused on LMS issues. This suggests 
that while first-year students struggle with technological 
issues, by the end of their second year of participation, 
students have successfully developed technological and 
problem-solving skills to overcome these issues. 


Teachers also provided additional perspective on some of 
the challenges that emerged in interviews. Specifically, 9th- 
grade teachers described the challenges of implementing 
STEM21 Academy courses in conjunction with other district 
requirements as sometimes “overwhelming,” occasionally 
resulting in decreased ability to implement all course com- 
ponents or activities. A few teachers, in particular teachers 
new to the program, described similar challenges with the 
technological demands of the courses. 
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PERCEIVED IMPACT OF THE FOUR CORE 
COMPONENTS OF BLENDED INSTRUCTION 


In the second stage of our interviews, students shared their 
perspectives of how each of the core blended instruction 
program components affected their learning. Each compo- 
nent is discussed separately. 


Technology-enhanced learning: Students reported that 
STEM21’s approach to blending online and in-person learn- 
ing supported their growth in each of the three categories 
previously identified: success with learning, engagement, 
and personal growth and development. 


A majority (77%) of 9th-grade students reported a posi- 
tive impact on learning and engagement, with comments 
like, “blended learning helps me learn,” or “I like it/love it,” 
and said blended learning is “good,” “great,” “different,” 
and “balanced.” The specific aspects of the technology-en- 
hanced curriculum most frequently described as contribut- 
ing to their learning and engagement included: 


e the use of the blended learning strategies in the ex- 
tended Challenge Project, 

e the ability to access their classroom and projects from 
any location via the online portal, 


e the incorporation of digital media tools (such as video 
editing and application design software) into the online 
environment, and 


¢ aclear focus on skill development. 


Although a large majority of students expressed positive 
impressions of technology-enhanced learning, almost a 
quarter of students (23%) said its impact was minimized 
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due to challenges faced in the process. They elaborated on 
some of the points raised previously, identifying a need for 
more frequent teacher guidance and more consistent use 
of textbooks and note taking. A small number of students 
attributed their challenges to an overall lack of interest in 
technology. 


Ninth-grade student feedback was supported by feedback 
from teachers and 10th-grade students. The majority of 
teachers described the positive impact of technology-en- 
hanced learning on student outcomes; they expressed that 
their students enjoyed collaborative learning, the openness 
and freedom to work independently at their own pace, and 
the array of opportunities to work with a variety of technolo- 
gies in a creative problem-solving process. 


The teachers who provided less positive feedback were less 
experienced with the program, and we hypothesize that they 
are likely to have also been most impacted by the technologi- 
cal challenges described previously. The majority of LOth- 
grade students did not report challenges with the technology- 
based environment, likely due to greater experience with 
technology and blended-learning strategies. 


Experiential learning: Data from interviews strongly 
indicate that both the smaller experiential meetings and the 
end-of-year Student Innovation Expo were viewed as posi- 
tive opportunities for students to broaden their educational 
perspectives in a less restrictive environment. 


A majority of students (66%) spoke positively about experien- 
tial meetings that took them from the comforts of a tradi- 
tional school setting into business environments and college 
campuses. Experiential meetings took a variety of forms and 
included hands-on learning with partner organizations—for 
example, environmental groups that brought students into 
the field to conduct data collection, or a computer company 
that helped students develop a business website. Students 
described these opportunities to do real-world and hands-on 
work as “inspiring,” critical to supporting their development 
of communication and presentation skills, and important 

to personal development. The majority of respondents also 
shared that the meetings enhanced their learning by offering 
opportunities to actively participate in scientific experiments 
and learn about innovative professional products. Students 
frequently described their participation in these meetings 

as improving their ability to bring fresh ideas back to the class- 
room and to incorporate these ideas into the design of their 
Challenge Projects. 


Diverging from the positive feedback were concerns regard- 
ing the mechanics of the experiential meetings. These 
individuals (less than a third of all respondents) noted a 
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lack of relationship between the meeting activities and the 
goal of the extended Challenge Project, a lack of connec- 
tion between activities and classwork, an over-emphasis on 
college tours, discomfort due to the required interactions 
with students from other districts, lack of interaction with 
other schools, and concerns over meetings canceled due 
to weather or other limitations. Teacher focus groups veri- 
fied some of these concerns. Although both students and 
teachers appreciated the function of group work in these 
experiential meetings, they expressed a need for these 
groups to include a balance of classmates and students 
from other schools, which would confer an optimal balance 
of emotional support with opportunities for challenge and 
new insights. These results suggest that some students 
and their teachers would benefit from more guidance with 
experiential learning meetings. 


The Student Innovation Expo is the culminating annual 
event at which STEM21 Academy student teams showcase 
their learning through their Challenge Projects. At the time 
of the 9th-grade interviews, the Expo had not yet taken 
place. Therefore, our results are based primarily on inter- 
viewer attendance at the Student Innovation Expo, sub- 
sequent field notes, and teacher and 10th-grade student 
input. 


Ninth-grade students, who had not yet participated in 

the Student Innovation Expo, expressed a high degree of 
anticipation for the event and commitment to and pride in 
their final Challenge Projects. Their enthusiasm was sup- 
ported by 10th-grade students and teachers, who expressed 
enthusiasm about the power of these opportunities to 
encourage collaboration, support personal and professional 
development, develop 21st century skills, and provide a 
central focus for all classroom activities. These discussions 
provided overwhelming evidence that the Student Innova- 
tion Expo and the related Challenge Project are viewed as 
critical to fostering student engagement, commitment, skill 
development, and content mastery. Maturation and previ- 
ous experience may account for differences in responses of 
9th-grade students and the more sophisticated responses 
of their older peers and teachers, who offered more explicit 
details. 


Few students and teachers gave negative feedback relating 
to the Student Innovation Expo. Among interviewed Qth- 
graders, 18% expressed negative concerns, such as a need 
to allocate more time for student presentations and a desire 
for more opportunities to view and learn from the work 

of other school districts. The findings were similar among 
10th-graders, with 14% providing at least some negative 
feedback. Like the 9th-graders, the most frequent concerns 
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pertained to lack of time for project preparations, or time 
allocated for presenting at the event. 


Digital portfolios: The digital portfolio, an online envi- 
ronment where students collect and share their work, was 
described positively by the majority (81%) of students. 
Recurring themes included the three categories discussed 
previously: enhanced learning, engagement, and personal 
and skill development. Students also praised the digital 
portfolio as a useful gauge of what they learned in an 
academic year and a resource with potential future value 
if treated as an expandable online resume for college and 
career applications. Digital portfolios were described as 
excellent tools for reflection, for providing insight into how 
an individual’s work measures up against the work of their 
peers, and for providing a means to display what a student 
has learned throughout their high school experience and 
beyond. Many students described the portfolio as a good 
frame of reference for track- 
ing their progress within a 
course as well as a useful tool 
for reviewing course content 
for exams and the end-of-year 
Challenge Project. 


Nearly a fifth of respondents 
(19%) expressed negative 
perceptions of the digital 
portfolio. The challenges they 
described included a lack of 
understanding of the role of the digital portfolio in the learn- 
ing process, problems navigating the links needed to access 
the portfolio, and a lack of teacher direction or knowledge 

in some schools. Several students stated that the portfolio 
was “busy work;” they felt the act of putting work in two 
places—the portfolio and the location where the work origi- 
nated—was redundant. It is worth noting that students that 
perceive the portfolio as nothing more than “putting work in 
two places” clearly did not have a strong understanding of 
the purpose of the portfolio. 


Results from teacher focus groups offer a possible explana- 
tion for the challenges expressed by students. A number 

of teachers explained that student buy into the portfolio 
requires understanding of the benefit it provides. A few 
hypothesized that more mature students, with a greater un- 
derstanding of the need to prepare for life after high school, 
generally demonstrated greater appreciation of the portfo- 
lio. Responses from both students and teachers revealed 
10th-graders as generally having a stronger understanding 
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of and appreciation for the portfolio. 


Proficiency assessments: STEM21 Academy’s Unit Per- 
formance Assessments (UPAS) measure students’ mastery 
of concepts and standards in each curricular unit, leading 
toward a final, end-of-course Challenge Project presented at 
the Student Innovation Expo. 


Student perceptions of the smaller-scale, in-class UPAs 
were varied and appeared to be related to their familiar- 
ity with the term “Unit Performance Assessment” as well 
as their awareness of pre-designated unit topics. Some 
students were not aware that they had participated in the 
assessments or remembered covering some topics but not 
others. It is worth noting that students indicated that the 
use of UPAs became less frequent as the end-of-year Expo 
approached. Students who did remember the proficiency 
assessments reflected positively about the role of the UPA in 
increasing collaboration. Students did not perceive that the 
UPAs given early in the year were 
relevant to the Challenge Project 
goals. 


..both students and teachers 
shared very positive perceptions 
of the long-term, proficiency- 
based Challenge Project. 


Teacher feedback supported stu- 
dent perceptions. Teacher focus 
group responses suggest that 
many teachers chose to use their 
own school’s or region’s teaching 
rubrics, or exchanged them inter- 
mittently with the UPAs. Teachers 
also indicated that the UPAs were not easily aligned with 
their course requirements, being either too basic or com- 
plex to meet their students’ needs. A number of teachers 
expressed concern that some of the UPAs lacked rigor and 
relevance, thus explaining why they had not implemented 
UPAs consistently. While their feedback on UPAs as a whole 
was mixed, teachers noted that specific UPA labs or assess- 
ment topics had a positive impact on student learning. 


In contrast, both students and teachers shared very positive 
perceptions of the long-term, proficiency-based Challenge 
Project. The vast majority of students perceived the Chal- 
lenge Project to be a highly valuable part of their learning 
experience. They overwhelmingly agreed that the project 
supported learning of course content and helped them 
develop skills in independent thinking and research, writing, 
presenting, oral communication, critical thinking, creativity, 
leadership, tolerance, and compromise. Students also de- 
scribed how these new skills helped them engage and learn 
more effectively in other subject areas. 
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Most students expressed deep investment in and commit- 
ment to solving the Challenge Project as well as pride in 
what they had created. They emphasized the “real-world” 
value of their projects; a number of respondents described 
inventing products or services that they felt would improve 
vital concerns related to the environment, personal safety, 
quality of life, health, or energy efficiency. Students ex- 
pressed ownership of specific project tasks, and recognized 
that successful completion of a Challenge Project required 
interdependence among students. Nearly all respondents 
stated that collaboration and teamwork were the most 
important skills learned while completing their Challenge 
Projects. 


Students also discussed the role of the Challenge Project in 
enhancing personal growth - in particular the development 
of self-confidence. Students and teachers attributed the 
growth in confidence to the opportunities they had to take 
on such challenges as interviewing industry professionals or 
public speaking. 


Students described relatively few problems with the Chal- 
lenge Project. Their most common concern was the time 
conflict between needing to complete end-of-year assign- 
ments and exams as well as their projects. A number of 
students perceived Challenge Projects to be aligned with 
course requirements, while others expressed concern that 
they fell behind in their studies due to the time require- 
ments of the Challenge Project. A small number of respon- 
dents mentioned lack of time for project development. 
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IMPACT ON AT-RISK AND UNDERSERVED 
STUDENTS 


Researchers decided that asking students or teachers 
about their ethnic background or socioeconomic status 
could limit the discussion or be seen as offensive. As a re- 
sult, no detailed demographic data is available on interview 
participants. We do know that during 2012-2013, 71.7% of 
all STEM21 Academy study participants were identified as 
underserved based on their socio-economic status, gender, 
and/or ethnic background. We can infer that there was a 
similar distribution among interview participants. Since 
our qualitative data indicates an extremely high positive 
perception of the impact of blended instruction on student 
learning, engagement, and personal development, we ex- 
pect that these results can be understood to apply to both 
underserved students and their peers. 


While we did not parse demographic data for this portion 
of the study, interviewer notes identified frequent refer- 
ences to technological challenges in the urban schools 
represented in our sample. Students in these schools were 
also far more likely to express challenges related to a lack 
of access to curriculum resources, stemming from a lack 
of computer access at home and less than ideal access 

at school. Additionally, interviewers noted lesser engage- 
ment in the interview process from some students in the 
urban schools; when these students were queried for more 
detailed responses to questions, they often stated that they 
didn’t know the answer. Interview data from urban schools 
generally contained less breadth and depth of information. 
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VARIATION OF IMPACT BY OTHER STUDENT 
CHARACTERISTICS 


In an attempt to understand whether STEM21 Academy’s 
blended instructional approach had a differential impact 
on students based on other personal characteristics, the 
research team reviewed interview and focus group results, 
field notes, and student demographic data. Student inter- 
view and teacher focus group results across the years of the 
study strongly suggest that students who benefit the most 
by STEM21 Academy participation are those who expressed 
interest in and readiness for a different kind of learning 
experience. Students who responded most positively to the 
program were: 


¢ not highly bound to the traditional instruction tech- 
niques of lecture and note taking, 


¢ not intensively concerned about the impact participa- 
tion in a non-Advanced Placement/honors course might 
have on their chances of college acceptance, 


¢ interested in working hard on a project and taking 
responsibility for their own learning, 


¢ comfortable with hands-on education, 


e interested in and comfortable with the use of technol- 
ogy in learning, and 


e less successful and/or comfortable in a traditional 
academic setting. 


Although these personal characteristics can be found 
across demographic groups, they are most likely to be 

found among students who have been less successful in a 
traditional school setting and may, therefore, occur more 
frequently among populations that are typically underserved 
by schools, including low-income students, English language 
learners, students with disabilities, and racial minorities. 


Supporting these findings, students and teachers alike 
perceived that those who were most likely to be success- 

ful in the STEM21 Academy were those who were willing to 
“take a risk” and “break out” of the traditional educational 
setting, interested in technology and hands-on learning, and 
open to committing to an intensive work experience. Both 
students and teachers describe the curriculum as “intense,” 
“a lot to learn,” and requiring students to take ownership of 
the material. 
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IMPORTANCE OF TECHNOLOGY AND OTHER 
RESOURCES 


Our interview and focus group results indicate that success- 
ful STEM21 Academy implementation requires adequate ac- 
cess to a variety of working technology, both at home and in 
the classroom. Additionally, the heavy technological require- 
ment of these courses requires that either the classroom 
teacher, or an easily accessible school staff member, have 
the technological ability to troubleshoot issues as needed. 
The most common technological concerns described by 
teachers and students included a lack of software compat- 
ibility between school computers and software required by 
the Academy courses and the need for a technology team to 
support access to the websites. Both teachers and students 
also described minor issues with the LMS, including design 
restrictions that limit flexibility and website links that may 
not be updated. 


It is also worth noting that participation in experiential 
meetings and the end-of-year Student Innovation Expo 
requires fiscal resources (e.g., funding for travel and pos- 
sible overnight stays), commitment of district leadership 
and parents (especially to support time spent away from 
the classroom), and willingness from staff, parents, and 
students to forgo AP/honors courses. 


FIDELITY OF IMPLEMENTATION 


As discussed in the study design section of this report, we 
rated implementation fidelity on several indicators for each 
of our focal areas: teacher professional development, class- 
room implementation, and program-wide implementation. 
Analysis of implementation fidelity allows us to see what is 
happening in a program as it is being implemented. The po- 
tential impact on classroom teachers, schools and program 
staff of variations in implementation can be large. We set 

a 65% threshold as our definition of adequate implementa- 
tion for each construct. Summary results are presented in 
Table 5. 


During 2011-2012, the threshold of 65% fidelity was 
achieved at the teacher, classroom, and program levels. 
During 2012-2013, fewer 9th-grade teachers than expected 
attended the summer professional development session. 
As a result, fidelity of teacher professional development 
was not achieved at the 9th-grade level. However, fidelity 
thresholds for classroom implementation and program 
implementation were met. It should be noted that ad- 
ditional professional development in the form of in class 
team teaching and coaching was provided, as well as online 
technical assistance. 
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IMPLICATIONS OF FIDELITY RESULTS 


We found that measuring implementation fidelity requires 
an extremely clearly defined program model. As our program 
staff developd a more clearly defined model, we improved 
the program in the process. In collecting fidelity data, we 
were able to clearly identify the STEM21 Academy’s four pri- 
mary program components (technology-enhanced learning, 
the digital portfolio, experiential learning, and performance 
assessments). Establishing this clear definition increased 
our ability to describe and explain the model to other 
researchers and program staff. It also helped us to identify 
appropriate data collection instruments to use for measur- 
ing implementation fidelity. 


The collection of implementation fidelity data was also in- 
strumental in helping program staff identify aspects of the 
program that were not being implemented as expected. This 
knowledge was used throughout the two years of the study 
to identify additional program support schools might need 
or to recommend changes in the implementation process. 
Using specific implementation data, program staff could 
identify teachers and other staff who were not performing 
as intended, using this information to provide support to 
these individuals, revise the program, or replace individuals 
who were not willing and able to implement the program as 
required. This access to data has greatly improved our 
ability to ensure that activities are implemented throughout 
participating schools as intended. Assessment of implemen- 
tation fidelity has been an extremely worthwhile experi- 


ence for the STEM21 team as a whole. Use of a rigorous 
assessment of implementation fidelity has improved our 
program, developed our research capacity, and strength- 
ened our awareness of the relationship between research 
and program development. We have found the process to 
be invaluable. 


Table 5: Fidelity of implementation results 


GOAL FOR % IMPLEMENTED GOAL FOR % IMPLEMENTED 
2011-12 WITH FIDELITY 2012-2013 WITH FIDELITY 
2011-12 2012-2013 
Teacher PD 65% of schools 80% of schools showed | 65% of schools with 58% of schools showed 
with high teacher high fidelity high scores for teacher _| high fidelity 
participation Above threshold participation Below threshold 
Classroom 65% of schools with 71% of schools showed _ | 65% of schools with 66% of schools showed 
Implementation Fidelity | high classroom fidelity high fidelity high scores for grade high fidelity 
Above threshold activities Above threshold 
Program Program has score of Score=4 Program has score Score=6 
> 2 out of 4 Above threshold > 4 out of 6 Above threshold 
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Our mixed-method study of the STEM21 Academy’s blended 
instructional model resulted in four major findings: 


e Participation in the STEM21 Academy significantly in- 
creased student achievement in science and marginally 
increased their 21st century and inquiry skills. 


e One year of STEM21 Academy exposure did not result 
in a significant increase in academic engagement. 


e« Students and teachers overwhelmingly reported posi- 
tive impacts on student learning through STEM21’s 
blended instructional approach. 


e Students were most positive about the experiential 
learning components of the program and extended 
Challenge Projects. There was a less favorable feed- 
back related to the program’s unit proficiency assess- 
ments. 


The first set of findings are consistent with other research 
that found that project-based learning, another student- 
centered approach, is effective in teaching 21st century 
skills (Moylan, 2008; Bell, 2010) and leads to increased 
achievement in science (Kanter & Konstantopoulos, 2010). 
We found that all students, and, in particular, underserved 
students in the STEM21 Academy, earned higher science 
achievement scores than students in the control group at 
the end of one year. This is consistent with prior literature 
that found that students in a blended learning environment 
perform better than students who learn the same material 
either exclusively online or in a traditional classroom setting 
alone (Heterick & Twigg, 2003). 
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Our results regarding student engagement are only par- 
tially supported by other studies. DeGeorge-Walker and 
colleagues found that a blended learning environment and 
information and communication technologies enhanced 
student engagement and achievement after one semester 
of an undergraduate course (DeGeorge-Walker, et al., 2010; 
Lou, et al., 2011). DeGeorge-Walker and colleagues used a 
qualitative, self-selected interview approach to understand 
engagement patterns, while we used a quantitative mea- 
sure. The differing data-gathering and analytic approaches 
could account for the difference in findings. Our own qualita- 
tive results, meanwhile, indicate a large increase in student 
engagement through STEM21 Academy participation. Both 
9th-grade students and teachers described all four com- 
ponents of the STEM21 Academy as enhancing student 
engagement in learning. The difference between our quan- 
titative and qualitative results regarding engagement may 
result from a difference in the areas of engagement that we 
explored. In our quantitative analysis, we examined more 
general indicators of engagement, such as doing class as- 
signments, consistently attending class, setting high expec- 
tations, and completing tasks. In our qualitative approach, 
we asked students and teachers specifically about the four 
instructional components of technology-enhanced learn- 
ing, experiential learning, digital portfolios, and proficiency 
assessments. 


The difference in the findings could also be attributed to 
the delayed effect of the treatment. Some scholars argue 
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that changes in personal factors, such as self-efficacy, take 
longer than changes in behavior (e.g., a choice of task, per- 
sistence, effort, and acquisition of skills) (Lou, et al., 2011; 
Schunk & Pajares, 2002). For instance, Lou et al. found 
that participation in a learning behavioral model showed a 
positive influence on students’ behavior in terms of cogni- 
tion and behavioral intentions, and they hypothesized that 
those changes in behavior would precede eventual changes 
in personal factors. 


Our interviews indicate that the specific elements of 
blended instruction that had the greatest impact were those 
associated with project-based and real-word learning (i.e., 
Challenge Projects, experiential learning). This mirrors the 
work of Schneider, Krajcik, Marx, and Soloway (2002) as 
well as the work of Hmelo-Silver, Duncan, and Chinn (2007), 
which demonstrates that project- and inquiry-based work in- 
creases achievement. STEM21 Academy’s Challenge Project 
provided opportunities for students to learn rigorous content 
and develop transferable skills through projects that were 
largely self-directed and collaborative. Students expressed 
preference for these extended projects when compared to 
shorter-term performance assessments. Although the Unit 
Performance Assessments were designed to scaffold the 
knowledge and skills required for the challenge project, 
greater value was placed on the Challenge Project. This 

was most likely due to the authentic nature of the project, 
the ability to learn in context, opportunities to connect with 
experts in the field, and chances to demonstrate expertise 
in an area of choice. 


Overall, our quantitative findings support the use of blended 
instruction to reduce the achievement gap for underserved 
students. This is consistent with other research that has 
found that 9th-grade, low-achieving students from minority 
backgrounds who used computer-assisted technology in the 
classroom outperformed their counterparts in classrooms 
without such technology (Shirvani, 2002). 


Our qualitative data reveals, though, that successful 
implementation of blended instruction requires students 
to access a variety of working technology, both at home 
and in the classroom. This finding points to the importance 
of providing support for the technology interventions that 
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take place in blended contexts (Graham, 2006). That is, 
parents and districts must have the necessary funding and 
infrastructure to support the technology requirements of a 
blended program, and, if necessary, provide at-home access 
for students who would not otherwise have access. This 
may be considered as a limitation of blended instruction, in 
that it may not be as cost-effective as completely online or 
completely face-to-face instructional approaches. However, 
access has become more ubiquitous and the majority of stu- 
dents, even in low socioeconomic settings, have availability 
to devices that can provide connectivity. A recent Pew Re- 
search Center survey found that teen students (ages 12-17) 
in urban centers have nearly the same access (94% access, 
74% mobile access) as their counterparts in suburban (96% 
access, 72% mobile access) and rural (99% access, 79% 
mobile access) environments (Madden, Lenhart, Duggan, 
Cortesi, & Gasser, 2013). The greatest disparities remain 
for those with low household income, regardless of urban- 
ity (89% access, 66% mobile access). Schools can leverage 
strategies like Bring Your Own Device to potentially remedi- 
ate and help leverage funds that provide access to students 
in a cost-effective manner (Costa, 2012). 


Conclusion 


The 21st century increasingly mandates the use of technol- 
ogy. As a future investment, it will be critical to provide tech- 
nology tools to students in all settings, particularly to those 
from low socioeconomic backgrounds. As teachers search for 
instructional strategies that have an impact on achievement, 
they should consider practices that increase student interac- 
tion and provide more opportunities for authentic learning. 
Technology-enhanced environments can be a powerful setting 
for leveraging authentic, student-centered learning opportuni- 
ties that connect students to their communities and provide 
extended learning challenges that result in gains in achieve- 
ment and engagement. 
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